Optimized network resource location

ABSTRACT

Resource requests made by clients of origin servers in a network are intercepted by reflector mechanisms and selectively reflected to other servers called repeaters. The reflectors select a best repeater from a set of possible repeaters and redirect the client to the selected best repeater. The client then makes the request of the selected best repeater. The resource is possibly rewritten to replace at least some of the resource identifiers contained therein with modified resource identifiers designating the repeater instead of the origin server.

1. FIELD OF THE INVENTION

[0001] This invention relates to replication of resources in computernetworks.

2. BACKGROUND OF THE INVENTION

[0002] The advent of global computer networks, such as the Internet,have led to entirely new and different ways to obtain information. Auser of the Internet can now access information from anywhere in theworld, with no regard for the actual location of either the user or theinformation. A user can obtain information simply by knowing a networkaddress for the information and providing that address to an appropriateapplication program such as a network browser.

[0003] The rapid growth in popularity of the Internet has imposed aheavy traffic burden on the entire network. Solutions to problems ofdemand (e.g., better accessibility and faster communication links) onlyincrease the strain on the supply. Internet Web sites (referred to hereas “publishers”) must handle ever-increasing bandwidth needs,accommodate dynamic changes in load, and improve performance for distantbrowsing clients, especially those overseas. The adoption ofcontent-rich applications, such as live audio and video, has furtherexacerbated the problem.

[0004] To address basic bandwidth growth needs, a Web publishertypically subscribes to additional bandwidth from an Internet serviceprovider (ISP), whether in the form of larger or additional “pipes” orchannels from the ISP to the publisher's premises, or in the form oflarge bandwidth commitments in an ISP's remote hosting servercollection. These increments are not always as fine-grained as thepublisher needs, and quite often lead times can cause the publisher'sWeb site capacity to lag behind demand.

[0005] To address more serious bandwidth growth problems, publishers maydevelop more complex and costly custom solutions. The solution to themost common need, increasing capacity, is generally based on replicationof hardware resources and site content (known as mirroring), andduplication of bandwidth resources. These solutions, however, aredifficult and expensive to deploy and operate. As a result, only thelargest publishers can afford them, since only those publishers canamortize the costs over many customers (and Web site hits).

[0006] A number of solutions have been developed to advance replicationand mirroring. In general, these technologies are designed for use by asingle Web site and do not include features that allow their componentsto be shared by many Web sites simultaneously.

[0007] Some solution mechanisms offer replication software that helpskeep mirrored servers up-to-date. These mechanisms generally operate bymaking a complete copy of a file system. One such system operates bytransparently keeping multiple copies of a file system in synch. Anothersystem provides mechanisms for explicitly and regularly copying filesthat have changed. Database systems are particularly difficult toreplicate, as they are continually changing. Several mechanisms allowfor replication of databases, although there are no standard approachesfor accomplishing it. Several companies offering proxy caches describethem as replication tools. However, proxy caches differ because they areoperated on behalf of clients rather than publishers.

[0008] Once a Web site is served by multiple servers, a challenge is toensure that the load is appropriately distributed or balanced amongthose servers. Domain name-server-based round-robin address resolutioncauses different clients to be directed to different mirrors.

[0009] Another solution, load balancing, takes into account the load ateach server (measured in a variety of ways) to select which servershould handle a particular request.

[0010] Load balancers use a variety of techniques to route the requestto the appropriate server. Most of those load-balancing techniquesrequire that each server be an exact replica of the primary Web site.Load balancers do not take into account the “network distance” betweenthe client and candidate mirror servers.

[0011] Assuming that client protocols cannot easily change, there aretwo major problems in the deployment of replicated resources. The firstis how to select which copy of the resource to use. That is, when arequest for a resource is made to a single server, how should the choiceof a replica of the server (or of that data) be made. We call thisproblem the “rendezvous problem”. There are a number of ways to getclients to rendezvous at distant mirror servers. These technologies,like load balancers, must route a request to an appropriate server, butunlike load balancers, they take network performance and topology intoaccount in making the determination.

[0012] A number of companies offer products which improve networkperformance by prioritizing and filtering network traffic. Proxy cachesprovide a way for client aggregators to reduce network resourceconsumption by storing copies of popular resources close to the endusers. A client aggregator is an Internet service provider or otherorganization that brings a large number of clients operating browsers tothe Internet. Client aggregators may use proxy caches to reduce thebandwidth required to serve web content to these browsers. However,traditional proxy caches are operated on behalf of Web clients ratherthan Web publishers.

[0013] Proxy caches store the most popular resources from allpublishers, which means they must be very large to achieve reasonablecache efficiency. (The efficiency of a cache is defined as the number ofrequests for resources which are already cached divided by the totalnumber of requests.)

[0014] Proxy caches depend on cache control hints delivered withresources to determine when the resources should be replaced. Thesehints are predictive, and are necessarily often incorrect, so proxycaches frequently serve stale data. In many cases, proxy cache operatorsinstruct their proxy to ignore hints in order to make the cache moreefficient, even though this causes it to more frequently serve staledata.

[0015] Proxy caches hide the activity of clients from publishers. Once aresource is cached, the publisher has no way of knowing how often it wasaccessed from the cache.

SUMMARY OF THE INVENTION

[0016] This invention provides a way for servers in a computer networkto off-load their processing of requests for selected resources bydetermining a different server (a “repeater”) to process those requests.The selection of the repeater can be made dynamically, based oninformation about possible repeaters.

[0017] If a requested resource contains references to other resources,some or all of these references can be replaced by references torepeaters.

[0018] Accordingly, in one aspect, this invention is a method ofprocessing resource requests in a computer network. First a client makesa request for a particular resource from an origin server, the requestincluding a resource identifier for the particular resource, theresource identifier sometimes including an indication of the originserver. Requests arriving at the origin server do not always include anindication of the origin server; since they are sent to the originserver, they do not need to name it. A mechanism referred to as areflector, co-located with the origin server, intercepts the requestfrom the client to the origin server and decides whether to reflect therequest or to handle it locally. If the reflector decides to handle therequest locally, it forwards it to the origin server, otherwise itselects a “best” repeater to process the request. If the request isreflected, the client is provided with a modified resource identifierdesignating the repeater.

[0019] The client gets the modified resource identifier from thereflector and makes a request for the particular resource from therepeater designated in the modified resource identifier.

[0020] When the repeater gets the client's request, it responds byreturning the requested resource to the client. If the repeater has alocal copy of the resource then it returns that copy, otherwise itforwards the request to the origin server to get the resource, and savesa local copy of the resource in order to serve subsequent requests.

[0021] The selection by the reflector of an appropriate repeater tohandle the request can be done in a number of ways. In the preferredembodiment, it is done by first pre-partitioning the network into “costgroups” and then determining which cost group the client is in. Next,from a plurality of repeaters in the network, a set of repeaters isselected, the members of the set having a low cost relative to the costgroup which the client is in. In order to determine the lowest cost, atable is maintained and regularly updated to define the cost betweeneach group and each repeater. Then one member of the set is selected,preferably randomly, as the best repeater.

[0022] If the particular requested resource itself can containidentifiers of other resources, then the resource may be rewritten(before being provided to the client). In particular, the resource isrewritten to replace at least some of the resource identifiers containedtherein with modified resource identifiers designating a repeaterinstead of the origin server. As a consequence of this rewritingprocess, when the client requests other resources based on identifiersin the particular requested resource, the client will make thoserequests directly to the selected repeater, bypassing the reflector andorigin server entirely.

[0023] Resource rewriting must be performed by reflectors. It may alsobe performed by repeaters, in the situation where repeaters “peer” withone another and make copies of resources which include rewrittenresource identifiers that designate a repeater.

[0024] In a preferred embodiment, the network is the Internet and theresource identifier is a uniform resource locator (URL) for designatingresources on the Internet, and the modified resource identifier is a URLdesignating the repeater and indicating the origin server (as describedin step B3 below), and the modified resource identifier is provided tothe client using a REDIRECT message. Note, only when the reflector is“reflecting” a request is the modified resource identifier providedusing a REDIRECT message.

[0025] In another aspect, this invention is a computer networkcomprising a plurality of origin servers, at least some of the originservers having reflectors associated therewith, and a plurality ofrepeaters.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects and advantages of the invention willbe apparent upon consideration of the following detailed description,taken in conjunction with the accompanying drawings, in which thereference characters refer to like parts throughout and in which:

[0027]FIG. 1 depicts a portion of a network environment according to thepresent invention; and

[0028] FIGS. 2-6 are flow charts of the operation of the presentinvention.

DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EXEMPLARY EMBODIMENTS

[0029] Overview

[0030]FIG. 1 shows a portion of a network environment 100 according tothe present invention, wherein a mechanism (reflector 108, described indetail below) at a server herein origin server 102) maintains and keepstrack of a number of partially replicated servers or repeaters 104 a,104 b, and 104 c. Each repeater 104 a, 104 b, and 104 c replicates someor all of the information available on the origin server 102 as well asinformation available on other origin servers in the network 100.Reflector 108 is connected to a particular repeater known as its“contact” repeater (“Repeater B” 104 b in the system depicted in FIG.1). Preferably each reflector maintains a connection with a singlerepeater known as its contact, and each repeater maintains a connectionwith a special repeater known as its master repeater (e.g., repeater 104m for repeaters 104 a, 104 b and 104 c in FIG. 1).

[0031] Thus, a repeater can be considered as a dedicated proxy serverthat maintains a partial or sparse mirror of the origin server 102, byimplementing a distributed coherent cache of the origin server. Arepeater may maintain a (partial) mirror of more than one origin server.In some embodiments, the network 100 is the Internet and repeatersmirror selected resources provided by origin servers in response toclients' HTTP (hypertext transfer protocol) and FTP (file transferprotocol) requests.

[0032] A client 106 connects, via the network 100, to origin server 102and possibly to one or more repeaters 104 a etc.

[0033] Origin server 102 is a server at which resources originate. Moregenerally, the origin server 102 is any process or collection ofprocesses that provide resources in response to requests from a client106. Origin server 102 can be any off-the-shelf Web server. In apreferred embodiment, origin server 102 is typically a Web server suchas the Apache server or Netscape Communications Corporation'sEnterprise™ server.

[0034] Client 106 is a processor requesting resources from origin server102 on behalf of an end user. The client 106 is typically a user agent(e.g., a Web browser such as Netscape Communications Corporation'sNavigator™) or a proxy for a user agent. Components other than thereflector 108 and the repeaters 104 a, 104 b, etc., may be implementedusing commonly available software programs. In particular, thisinvention works with any HTTP client (e.g., a Web browser), proxy cache,and Web server. In addition, the reflector 108 might be fully integratedinto the data server 112 (for instance, in a Web Server). Thesecomponents might be loosely integrated based on the use of extensionmechanisms (such as so-called add-in modules) or tightly integrated bymodifying the service component specifically to support the repeaters.

[0035] Resources originating at the origin server 102 may be static ordynamic. That is, the resources may be fixed or they may be created bythe origin server 102 specifically in response to a request Note thatthe terms “static” and “dynamic” are relative, since a static resourcemay change at some regular, albeit long, interval.

[0036] Resource requests from the client 106 to the origin server 102are intercepted by reflector 108 which for a given request eitherforwards the request on to the origin server 102 or conditionallyreflects it to some repeater 104 a, 104 b, etc. in the network 100. Thatis, depending on the nature of the request by the client 106 to theorigin server 102, the reflector 108 either serves the request locally(at the origin server 102), or selects one of the repeaters (preferablythe best repeater for the job) and reflects the request to the selectedrepeater. In other words, the reflector 108 causes requests forresources from origin server 102, made by client 106, to be eitherserved locally by the origin server 102 or transparently reflected tothe best repeater 104 a, 104 b, etc. The notion of a best repeater andthe manner in which the best repeater is selected are described indetail below.

[0037] Repeaters 104 a, 104 b, etc. are intermediate processors used toservice client requests thereby improving performance and reducing costsin the manner described herein. Within repeaters 104 a, 104 b, etc., areany processes or collections of processes that deliver resources to theclient 106 on behalf of the origin server 102. A repeater may include arepeater cache 110, used to avoid unnecessary transactions with theorigin server 102.

[0038] The reflector 108 is a mechanism, preferably a software program,that intercepts requests that would normally be sent directly to theorigin server 102. While shown in the drawings as separate components,the reflector 108 and the origin server 102 are typically co-located,e.g., on a particular system such as data server 112. (As discussedbelow, the reflector 108 may even be a “plug in” module that becomespart of the origin server 102.

[0039]FIG. 1 shows only a part of a network 100 according to thisinvention. A complete operating network consists of any number ofclients, repeaters, reflectors, and origin servers. Reflectorscommunicate with the repeater network, and repeaters in the networkcommunicate with one another.

[0040] Uniform Resource Locators

[0041] Each location in a computer network has an address which cangenerally be specified as a series of names or numbers. In order toaccess information, an address for that information must be known. Forexample, on the World Wide Web (“the Web”) which is a subset of theInternet, the manner in which information address locations are providedhas been standardized into Uniform Resource Locators (URLs). URLsspecify the location of resources (information, data files, etc.) on thenetwork.

[0042] The notion of URLs becomes even more useful when hypertextdocuments are used. A hypertext document is one which includes, withinthe document itself, links (pointers or references) to the documentitself or to other documents. For example, in an on-line legal researchsystem, each case may be presented as a hypertext document. When othercases are cited, links to those cases can be provided. In this way, whena person is reading a case, they can follow cite links to read theappropriate parts of cited cases.

[0043] In the case of the Internet in general and the World Wide Webspecifically, documents can be created using a standardized form knownas the Hypertext Markup Language (HTML). In HTML, a document consists ofdata (text, images, sounds, and the like), including links to othersections of the same document or to other documents. The links aregenerally provided as URLs, and can be in relative or absolute form.Relative URLs simply omit the parts of the URL which are the same as forthe document including the link, such as the address of the document(when linking to the same document), etc. In general, a browser programwill fill in missing parts of a URL using the corresponding parts fromthe current document, thereby forming a fully formed URL including afully qualified domain name, etc.

[0044] A hypertext document may contain any number of links to otherdocuments, and each of those other documents may be on a differentserver in a different part of the world. For example, a document maycontain links to documents in Russia, Africa, China and Australia. Auser viewing that document at a particular client can follow any of thelinks transparently (i.e., without knowing where the document beinglinked to actually resides). Accordingly, the cost (in terms of time ormoney or resource allocation) of following one link versus another maybe quite significant.

[0045] URLs generally have the following form (defined in detail in T.Berners-Lee et al, Uniform Resource Locators (URL), Network WorkingGroup, Request for Comments: 1738, Category: Standards Track, December1994, located at “http://ds.internic.net/rfc/rfc1738.txt”, which ishereby incorporated herein by reference):

[0046] schemee://host[part]/url-path

[0047] where “scheme” can be a symbol such as “file” (for a file on thelocal system), “ftp” (for a file on an anonymous FTP file server),“http” (for a file on a file on a Web server), and “telnet” (for aconnection to a Telnet-based service). Other schemes, can also be usedand new schemes are added every now and then. The port number isoptional, the system substituting a default port number (depending onthe scheme) if none is provided. The “host” field maps to a particularnetwork address for a particular computer. The “url-path” is relative tothe computer specified in the “host” field. A url-path is typically, butnot necessarily, the pathname of a file in a web server directory.

[0048] For example, the following is a URL identifying a file “F” in thepath “A/B/C” on a computer at “www.uspto.gov”:

[0049] http://www.uspto.gov/A/B/C/F

[0050] In order to access the file “F” (the resource) specified by theabove URL, a program (e.g., a browser) running on a user's computer(i.e., a client computer) would have to first locate the computer (i.e.,a server computer) specified by the host name. I.e., the program wouldhave to locate the server “www.uspto.gov”. To do this, it would access aDomain Name Server (DNS), providing the DNS with the host name(“www.uspto.gov”). The DNS acts as a kind of centralized directory forresolving addresses from names. If the DNS determines that there is a(remote server) computer corresponding to the name “www.uspto.gov”, itwill provide the program with an actual computer network address forthat server computer. On the Internet this is called an InternetProtocol (or IP) address and it has the form “123.345.456.678”. Theprogram on the user's (client) computer would then use the actualaddress to access the remote (server) computer.

[0051] The program opens a connection to the HTTP server. (Web server)on the remote computer “www.uspto.gov” and uses the connection to send arequest message to the remote computer (using the HTTP scheme). Themessage is typically an HTTP GET request which includes the url-path ofthe requested resource, “A/B/C/F”. The HTTP server receives the requestand uses it to access the resource specified by the url-path “A/B/C/F”.The server returns the resource over the same connection.

[0052] Thus, conventionally HTTP client requests for Web resources at anorigin server 102 are processed as follows (see FIG. 2) (This is adescription of the process when no reflector 108 is installed.):

[0053] A1. A browser (e.g., Netscape's Navigator) at the client receivesa resource identifier (i.e., a URL) from a user.

[0054] A2. The browser extracts the host (origin server) name from theresource identifier, and uses a domain name server (DNS) to look up thenetwork (IP) address of the corresponding server. The browser alsoextracts a port number, if one is present, or uses a default port number(the default port number for http requests is 80).

[0055] A3. The browser uses the server's network address and port numberto establish a connection between the client 106 and the host or originserver 102.

[0056] A4. The client 106 then sends a (GET) request over the connectionidentifying the requested resource.

[0057] A5. The origin server 102 receives the request and

[0058] A6. locates or composes the corresponding resource.

[0059] A7. The origin server 102 then sends back to the client 106 areply containing the requested resource (or some form of error indicatorif the resource is unavailable). The reply is sent to the client overthe same connection as that on which the request was received from theclient.

[0060] A8. The client 106 receives the reply from the origin server 102.

[0061] There are many variations of this basic model. For example, inone variation, instead of providing the client with the resource, theorigin server can tell the client to re-request the resource by anothername. To do so, in A7 the server 102 sends back to the client 106 areply called a “REDIRECT” which contains a new URL indicating the othername. The client 106 then repeats the entire sequence, normally withoutany user intervention, this time requesting the resource identified bythe new URL.

[0062] System Operation

[0063] In this invention reflector 108 effectively takes the place of anordinary Web server or origin server 102. The reflector 108 does this bytaking over the origin server's IP address and port number. In this way,when a client tries to connect to the origin server 102, it willactually connect to the reflector 108. The original Web server (ororigin server 102) must then accept requests at a different network (IP)address, or at the same IP address but on a different port number. Thus,using this invention, the server referred to in A3-A7 above is actuallya reflector 108.

[0064] Note that it is also possible to leave the origin server'snetwork address as it is and to let the reflector run at a differentaddress or on a different port. In this way the reflector does notintercept requests sent to the origin server, but can still be sentrequests addressed specifically to the reflector. Thus the system can betested and configured without interrupting its normal operation.

[0065] The reflector 108 supports the processing as follows (see FIG.3): upon receipt of a request,

[0066] B1 The reflector 108 analyzes the request to determine whether ornot to reflect the request. To do this, first the reflector determineswhether the sender (client 106) is a browser or a repeater. Requestsissued by repeaters must be served locally by the origin server 102.This determination can be made by looking up the network (IP) address ofthe sender in a list of known repeater network (IP) addresses.Alternatively, this determination could be made by attaching informationto a request to indicate that the request is from a specific repeater,or repeaters can request resources from a special port other than theone used for ordinary clients.

[0067] B2 If the request is not from a repeater, the reflector looks upthe requested resource in a table (called the “rule base”) to determinewhether the resource requested is “repeatable”. Based on thisdetermination, the reflector either reflects the request (B3, describedbelow) or serves the request locally (B4, described below).

[0068] The rule base is a list of regular expressions and associatedattributes. (Regular expressions are well-known in the field of computerscience. A small bibliography of their use is found in Aho, et al.,“Compilers, Principles, techniques and tools”, Addison-Wesley, 1986, pp.157-158.) The resource identifier (URL) for a given request is looked upin the rule base by matching it sequentially with each regularexpression. The first match identifies the attributes for the resource,namely repeatable or local. If there is no match in the rule base, adefault attribute is used. Each reflector has its own rule base, whichis manually configured by the reflector operator.

[0069] B3. To reflect a request, (to serve a request locally go to B4),as shown in FIG. 4, the reflector determines (B3-1) the best repeater toreflect the request to, as described in detail below. The reflector thencreates (B3-2) a new resource identifier (URL) (using the requested URLand the best repeater) that identifies the same resource at the selectedrepeater.

[0070] It is necessary that the reflection step create a single URLcontaining the URL of the original resource, as well as the identity ofthe selected repeater. A special form of URL is created to provide thisinformation. This is done by creating a new URL as follows:

[0071] D1. Given a repeater name, scheme, origin server name and path,create a new URL. If the scheme is “http”, the preferred embodiment usesthe following format

[0072] http://<repeater>/<server>/<path>

[0073]  If the form used is other than “http”, the preferred embodimentuses the following format:

[0074] http://<repeater>/<server>@proxy=<scheme>@/<path>

[0075]  The reflector can also attach a MIME type to the request, tocause the repeater to provide that MIME type with the result. This isuseful because many protocols (such as FTP) do not provide a way toattach a MIME type to a resource. The format is

[0076] http://<repeater>/<server>@proxy=<scheme>@/<path>

[0077]  This URL is interpreted when received by the repeater.

[0078] The reflector then sends (B3-3) a REDIRECT reply containing thisnew URL to the requesting client. The HTTP REDIRECT command allows thereflector to send the browser a single URL to retry the request

[0079] B4. To serve a request locally, the request is sent by thereflector to (“forwarded to”) the origin server 102. In this mode, thereflector acts as a reverse proxy server. The origin server 102processes the request in the normal manner (A5-A7). The reflector thenobtains the origin server's reply to the request which it inspects todetermine if the requested resource is an HTML document, i.e., whetherthe requested resource is one which itself contains resourceidentifiers.

[0080] B5. If the resource is an HTML document then the reflectorrewrites the HTML document by modifying resource identifiers (URLs)within it, as described below. The resource, possibly as modified byrewriting, is then returned in a reply to the requesting client 106.

[0081] If the requesting client is a repeater, the reflector maytemporarily disable any cache-control modifiers which the origin serverattached to the reply. These disabled cache-control modifiers are laterre-enabled when the content is served from the repeater. This mechanismmakes it possible for the origin server to prevent resources from beingcached at normal proxy caches, without affecting the behavior of thecache at the repeater.

[0082] B6. Whether the request is reflected or handled locally, detailsabout the transaction, such as the current time, the address of therequester, the URL requested, and the type of response generated, arewritten by the reflector to a local log file.

[0083] By using a rule base (B2), it is possible to selectively reflectresources. There are a number of reasons that certain particularresources cannot be effectively repeated (and therefore should not bereflected), for instance:

[0084] the resource is composed uniquely for each request;

[0085] the resource relies on a so-called cookie (browsers will not sendcookies to repeaters with different domain names);

[0086] the resource is actually a program (such as a Java applet) thatwill run on the client and that wishes to connect to a service (Javarequires that the service be running on the same machine that providedthe applet).

[0087] In addition, the reflector 108 can be configured so that requestsfrom certain network addresses (e.g., requests from clients on the samelocal area network as the reflector itself) are never reflected. Also,the reflector may choose not to reflect requests because the reflectoris exceeding its committed aggregate information rate, as describedbelow.

[0088] A request which is reflected is automatically mirrored at therepeater when the repeater receives and processes the request.

[0089] The combination of the reflection process described here and thecaching process described below effectively creates a system in whichrepeatable resources are migrated to and mirrored at the selectedreflector, while non-repeatable resources are not mirrored.

[0090] Alternate Approach

[0091] Placing the origin server name in the reflected URL is generallya good strategy, but it may be considered undesirable for aesthetic or(in the case, e.g., of cookies) certain technical reasons.

[0092] It is possible to avoid the need for placing both the repeatername and the server name in the URL. Instead, a “family” of names may becreated for a given origin server, each name identifying one of therepeaters used by that server.

[0093] For instance, if www.example.com is the origin server, names forthree repeaters might be created:

[0094] wr1.example.com

[0095] wr2.example.com

[0096] wr3.example.com

[0097] The name “wr1.example.com” would be an alias for repeater 1,which might also be known by other names such as“wr1.anotherExample.com” and “wr1.example.edu”.

[0098] If the repeater can determine by which name it was addressed, itcan use this information (along with a table that associates repeateralias names with origin server names) to determine which origin serveris being addressed. For instance, if repeater 1 is addressed aswr1.example.com, then the origin server is “www.example.com”; if it isaddressed as “wr1.anotherExample.com”, then the origin server is“www.anotherExample.com”.

[0099] The repeater can use two mechanisms to determine by which aliasit is addressed:

[0100] 1. Each alias can be associated with a different IP address.Unfortunately, this solution does not scale well, as IP addresses arecurrently scarce, and the number of IP addresses required grows as theproduct of origin servers and repeaters.

[0101] 2. The repeater can attempt to determine the alias name used byinspecting the “host:” tag in the HTTP header of the request.Unfortunately, some old browsers still in use do not attach the “host:”tag to a request. Reflectors would need to identify such browsers (thebrowser identity is a part of each request) and avoid this form ofreflection.

[0102] How a Repeater Handles a Request

[0103] When a browser receives a REDIRECT response (as produced in B3),it reissues a request for the resource using the new resource identifier(URL) (A1-A5). Because the new identifier refers to a repeater insteadof the origin server, the browser now sends a request for the resourceto the repeater which processes a request as follows, with reference toFIG. 5:

[0104] C1. First the repeater analyzes the request to determine thenetwork address of the requesting client and the path of the resourcerequested. Included in the path is an origin server name (as describedabove with reference to B3).

[0105] C2. The repeater uses an internal table to verify that the originserver belongs to a known “subscriber”. A subscriber is an entity (e.g.,a company) that publishes resources (e.g., files) via one or more originservers. When the entity subscribes, it is permitted to utilize therepeater network. The subscriber tables described below include theinformation that is used to link reflectors to subscribers.

[0106] If the request is not for a resource from a known subscriber, therequest is rejected. To reject a request, the repeater returns a replyindicating that the requested resource does not exist.

[0107] C3. The repeater then determines whether the requested resourceis cached locally. If the requested resource is in the repeater's cacheit is retrieved. On the other hand, if a valid copy of the requestedresource is not in the repeater's cache, the repeater modifies theincoming URL, creating a request that it issues directly to theoriginating reflector which processes it (as in B1-B6). Because thisrequest to the originating reflector is from a repeater, the reflectoralways returns the requested resource rather than reflecting therequest. (Recall that reflectors always handle requests from repeaterslocally.) If the repeater obtained the resource from the origin server,the repeater then caches the resource locally.

[0108] If a resource is not cached locally, the cache can query its“peer caches” to see if one of them contains the resource, before or atthe same time as requesting the resource from the reflector/originserver. If a peer cache responds positively in a limited period of time(preferably a small fraction of a second), the resource will beretrieved from the peer cache.

[0109] C4. The repeater then constructs a reply including the requestedresource (which was retrieved from the cache or from the origin server)and sends that reply to the requesting client.

[0110] C5. Details about the transaction, such as the associatedreflector, the current time, the address of the requester, the URLrequested, and the type of response generated, are written to a locallog file at the repeater.

[0111] Note that the bottom row of FIG. 2 refers to an origin server, ora reflector, or a repeater, depending on what the URL in step Alidentifies.

[0112] Selecting the Best Repeater

[0113] If the reflector 108 determines that it will reflect the request,it must then select the best repeater to handle that request (asreferred to in step B3-1). This selection is performed by the BestRepeater Selector (BRS) mechanism described here.

[0114] The goal of the BRS is to select, quickly and heuristically, anappropriate repeater for a given client given only the network addressof the client. An appropriate repeater is one which is not too heavilyloaded and which is not too far from the client in terms of some measureof network distance. The mechanism used here relies on specific,compact, pre-computed data to make a fast decision. Other, dynamicsolutions can also be used to select an appropriate repeater.

[0115] The BRS relies on three pre-computed tables, namely the GroupReduction able, the Link Cost Table, and the Load Table. These threetables (described below) are computed off-line and downloaded to eachreflector by its contact in the repeater network.

[0116] The Group Reduction Table places every network address into agroup, with the goal that addresses in a group share relative costs, sothat they would have the same best repeater under varying conditions(i.e., the BRS is invariant over the members of the group).

[0117] The Link Cost Table is a two dimensional matrix which specifiesthe current cost between each repeater and each group. Initially, thelink cost between a repeater and a group is defined as the “normalizedlink cost” between the repeater and the group, as defined below. Overtime, the table will be updated with measurements which more accuratelyreflect the relative cost of transmitting a file between the repeaterand a member of the group. The format of the Link Cost Table is <GroupID><Group ID><link cost>, where the Group ID's are given as AS numbers.

[0118] The Load Table is a one dimensional table which identifies thecurrent load at each repeater. Because repeaters may have differentcapacities, the load is a value that represents the ability of a givenrepeater to accept additional work. Each repeater sends its current loadto a central master repeater at regular intervals, preferably at leastapproximately once a minute. The master repeater broadcasts the LoadTable to each reflector in the network, via the contact repeater.

[0119] A reflector is provided entries in the Load Table only forrepeaters which it is assigned to use. The assignment of repeaters toreflectors is performed centrally by a repeater network operator at themaster repeater. This assignment makes it possible to modify the servicelevel of a given reflector. For instance, a very active reflector mayuse many repeaters, whereas a relatively inactive reflector may use fewrepeaters.

[0120] Tables may also be configured to provide selective repeaterservice to subscribers in other ways, e.g., for their clients inspecific geographic regions, such as Europe or Asia.

[0121] Measuring Load

[0122] In the presently preferred embodiments, repeater load is measuredin two dimensions, namely

[0123] 1. requests received by the repeater per time interval (RRPT),and

[0124] 2. bytes sent by the repeater per time interval (BSPT).

[0125] For each of these dimensions, a maximum capacity setting is set.The maximum capacity indicates the point at which the repeater isconsidered to be fully loaded. A higher RRPT capacity generallyindicates a faster processor, whereas a higher BSPT capacity generallyindicates a wider network pipe. This form of load measurement assumesthat a given server is dedicated to the task of repeating.

[0126] Each repeater regularly calculates its current RRPT and BSPT, byaccumulating the number of requests received and bytes sent over a shorttime interval. These measurements are used to determine the repeater'sload in each of these dimensions. If a repeater's load exceeds itsconfigured capacity, an alarm message is sent to the repeater networkadministrator.

[0127] The two current load components are combined into a single valueindicating overall current load. Similarly, the two maximum capacitycomponents are combined into a single value indicating overall maximumcapacity. The components are combined as follows:

current-load=B×current RRPT+(1−B)×current BSPT

max-load =B×max RRPT+(1−B)×max BSPT

[0128] The factor B, a value between 0 and 1, allows the relativeweights of RRPT and BSPT to be adjusted, which favors consideration ofeither processing power or bandwidth.

[0129] The overall current load and overall maximum capacity values areperiodically sent from each repeater to the master repeater, where theyare aggregated in the Load Table, a table summarizing the overall loadfor all repeaters. Changes in the Load Table are distributedautomatically to each reflector.

[0130] While the preferred embodiment uses a two-dimensional measure ofrepeater load, any other measure of load can be used.

[0131] Combining Link Costs and Load

[0132] The BRS computes the cost of servicing a given client from eacheligible repeater. The cost is computed by combining the availablecapacity of the candidate repeater with the cost of the link betweenthat repeater and the client. The link cost is computed by simplylooking it up in the Link Cost table.

[0133] The cost is determined using the following formula:

thresbold=K*max-load

capacity=max(max-load−current-load, e)

capacity=min(capacity, threshold)

cost=link-cost*threshold capacity

[0134] In this formula, e is a very small number (epsilon) and K is atuning factor initial set to 0.5. This formula causes the cost to agiven repeater to be increased, at a rate defined by K, if its capacityfalls below a configurable threshold.

[0135] Given the cost of each candidate repeater, the BRS selects allrepeaters within a delta factor of the best score. From this set, theresult is selected at random.

[0136] The delta factor prevents the BRS from repeatedly selecting asingle repeater when scores are similar. It is generally requiredbecause available information about load and link costs loses accuracyover time. This factor is tunable.

[0137] Best Repeater Selector (BRS)

[0138] The BRS operates as follows, with reference to FIG. 6:

[0139] Given a client network address and the three tables describedabove:

[0140] E1. Determine which group the client is in using the GroupReduction Table.

[0141] E2. For each repeater in the Link Cost Table and Load Table,determine that repeater's combined cost as follows:

[0142] E2a: Determine the maximum and current load on the repeater(using the Load Table).

[0143] E2b. Determine the link cost between the repeater and theclient's group (using the Link Cost Table).

[0144] E2c. Determine the combined cost as described above.

[0145] E3. Select a small set of repeaters with the lowest cost.

[0146] E4. Select a random member from the set

[0147] Preferably the results of the BRS processing are maintained in alocal cache at the reflector 108. Thus, if the best repeater hasrecently been determined for a given client (i.e., for a given networkaddress), that best repeater can be reused quickly without beingre-determined. Since the calculation described above is based onstatically, pre-computed tables, if the tables have not changed thenthere is no need to re-determine the best repeater.

[0148] Determining the Group Reduction and Link Cost Tables

[0149] The Group Reduction Table and Link Cost Table used in BRSprocessing are created and regularly updated by an independent procedurereferred to herein as NetMap. The NetMap procedure is run by executingseveral phases (described below) as needed.

[0150] The term Group is used here to refers to an IP “address group”.

[0151] The term Repeater Group refers to a Group that contains the IPaddress of a repeater.

[0152] The term link cost refers to a statically determined cost fortransmitting data between two Groups. In a presently preferredimplementation, this is the minimum of the sums of the costs of thelinks along each path between them. The link costs of primary concernhere are link costs between a Group and a Repeater Group.

[0153] The term relative link cost refers to the link cost relative toother link costs for the same Group which is calculated by subtractingthe minimum link cost from a Group to any Repeater Group from each ofits link costs to a Repeater Group. The term Cost Set refers to a set ofGroups that are equivalent in regard to Best Repeater Selection. Thatis, given the information available, the same repeater would be selectedfor any of them.

[0154] The NetMap procedure first processes input files to create aninternal database called the Group Registry. These input files describegroups, the IP addresses within groups, and links between groups, andcome a variety of sources, including publicly available Internet RoutingRegistry (IRR) databases, BGP router tables, and probe services that arelocated at various points around the Internet and use publicly availabletools (such as “traceroute”) to sample data paths. Once this processingis complete, the Group Registry contains essential information used forfurther processing, namely (1) the identity of each group, (2) the setof IP addresses in a given group, (3) the presence of links betweengroups indicating paths over which information may travel, and (4) thecost of sending data over a given link.

[0155] The following processes are then performed on the Group Registryfile.

[0156] Calculate Repeater Group link costs

[0157] The NetMap procedure calculates a “link cost” for transmission ofdata between each Repeater Group and each Group in the Group Registry.This overall link cost is defined as the minimum cost of any pathbetween the two groups, where the cost of a path is equal to the sum ofthe costs of the individual links in the path. The link cost algorithmpresented below is essentially the same as algorithm #562 from ACMjournal Transactions on Mathematical Software: “Shortest Path From aSpecific Node to All Other Nodes in a Network” by U. Pape, ACM TOMS 6(1980) pp. 450-455, http://www.netlib.org/toms/562.

[0158] In this processing, the term Repeater Group refers to a Groupthat contains the IP address of a repeater. A group is a neighbor ofanother group if the Group Registry indicates that there is a linkbetween the two groups.

[0159] For each target Repeater Group T:

[0160] Initialize the link cost between T and itself to zero.

[0161] Initialize the link cost between T and every other Group toinfinity.

[0162] Create a list L that will contain Groups that are equidistantfrom the target Repeater Group T.

[0163] Initialize the list L to contain just the target Repeater Group Titself

[0164] While the list L is not empty:

[0165] Create an empty list L′ of neighbors of members of the list L.

[0166] For each Group G in the list L:

[0167] For each Group N that is a neighbor of G:

[0168] Let cost refer to the sum of the link cost between T and G, andthe link cost between G and N.

[0169] The cost between T and G was determined in the previous pass ofthe algorithm; the link cost between G and N is from the Group Registry.

[0170] If cost is less than the link cost between T and N:

[0171] Set the link cost between T and N to cost.

[0172] Add N to L′ if it is not already on it.

[0173] Set L to L′.

[0174] Calculate Cost Sets

[0175] A Cost Set is a set of Groups that are equivalent with respect toBest Repeater Selection. That is, given the information available, thesame repeater would be selected for any of them.

[0176] The “cost profile” of a Group G is defined herein as the set ofcosts between G and each Repeater. Two cost profiles are said to beequivalent if the values in one profile differ from the correspondingvalues in the other profile by a constant amount.

[0177] Once a client Group is known, the Best Repeater Selectionalgorithm relies on the cost profile for information about the Group. Iftwo cost profiles are equivalent, the BRS algorithm would select thesame repeater given either profile.

[0178] A Cost Set is then a set of groups that have equivalent costprofiles.

[0179] The effectiveness of this method can be seen, for example, in thecase where all paths to a Repeater from some Group A pass through someother Group B. The two Groups have equivalent cost profiles (and aretherefore in the same Cost Set) since whatever Repeater is best forGroup A is also going to be best for Group B, regardless of what path istaken between the two Groups.

[0180] By normalizing cost profiles, equivalent cost profiles can bemade identical. A normalized cost profile is a cost profile in which theminimum cost has the value zero. A normalized cost profile is computedby finding the minimum cost in the profile, and subtracting that valuefrom each cost in the profile.

[0181] Cost Sets are then computed using the following algorithm:

[0182] For each Group G:

[0183] Calculate the normalized cost profile for G

[0184] Look for a Cost Set with the same normalized cost profile.

[0185] If such as set is found, add G to the existing Cost Set;

[0186] otherwise, create a new Cost Set with the calculated normalizedcost profile, containing only G.

[0187] The algorithm for finding Cost Sets employs a hash table toreduce the time necessary to determine whether the desired Cost Setalready exists. The hash table uses a hash value computed from costprofile of G.

[0188] Each Cost Set is then numbered with a unique Cost Sent Indexnumber. Cost Sets are then used in a straightforward manner to generatethe Link Cost Table, which gives the cost from each Cost Set to eachRepeater.

[0189] As described below, the Group Reduction Table maps every IPaddress to one of these Cost Sets.

[0190] Build IP Map

[0191] The IP Map is a sorted list of records which map IP addressranges to Link Cost Table keys. The format of the IP map is:

[0192] <base IP address><max IP address><Link Cost Table key>

[0193] where IP addresses are presently represented by 32-bit integers.The entries are sorted by descending base address, and by ascendingmaximum address among equal base addresses, and by ascending Link CostTable key among equal base addresses and maximum addresses. Note thatranges may overlap.

[0194] The NetMap procedure generates an intermediate IP map containinga map between IP address ranges and Cost Set numbers as follows:

[0195] For each Cost Set S:

[0196] For each Group G in S:

[0197] For each IP address range in G:

[0198] Add a triple (low address, high address, Cost Set number of S) tothe IP map.

[0199] The IP map file is then sorted by descending base address, and byascending maximum address among equal base addresses, and by ascendingCost Set number among equal base addresses and maximum addresses. Thesort order for the base address and maximum address minimizes the timeto build the Group Reduction Table and produces the proper results foroverlapping entries.

[0200] Finally, the NetMap procedure creates the Group Reduction Tableby processing the sorted IP map. The Group Reduction Table maps IPaddresses (specified by ranges) into Cost Set numbers. Specialprocessing of the IP map file is required in order to detect overlappingaddress ranges, and to merge adjacent address ranges in order tominimize the size of the Group Reduction Table.

[0201] An ordered list of address range segments is maintained, eachsegment consisting of a base address B and a Cost Set number N, sortedby base address B. (The maximum address of a segment is the base addressof the next segment minus one.)

[0202] The following algorithm is used:

[0203] Initialize the list with the elements [−infinity, NOGROUP],[+infinity, NOGROUP].

[0204] For each entry in the IP map, in sorted order, consisting of (b,m, s),

[0205] Insert (b, m, s) in the list (recall that IP map entries are ofthe form (low address, high address Cost Set number of S))

[0206] For each reserved LAN address range (b, m):

[0207] Insert (b, m, LOCAL) in the list.

[0208] For each Repeater at address a:

[0209] Insert (a, a, REPEATER) in the list.

[0210] For each segment S in the ordered list:

[0211] Merge S with following segments with the same Cost Set

[0212] Create a Group Reduction Table entry with base address from thebase address of S,

[0213] max address=next segment's base—1,

[0214] group ID=Cost Set number of S.

[0215] A reserved LAN address range is an address range reserved for useby LANs which should not appear as a global Internet address. LOCAL is aspecial Cost Set index different from all others, indicating that therange maps to a client which should never be reflected. REPEATER is aspecial Cost Set index different from all others, indicating that theaddress range maps to a repeater. NOGROUP is a special Cost Set indexdifferent from all others, indicating that this range of addresses hasno known mapping.

[0216] Given (B, M, N), insert an entry in the ordered address list asfollows:

[0217] Find the last segment (AB, AN) for which AB is less than or equalto B.

[0218] If AB is less than B, insert a new segment (B, N) after (AB, AN).

[0219] Find the last segment (YB, YN) for which YB is less than or equalto M.

[0220] Replace by (XB, N) any segment (XB, NOGROUP) for which XB isgreater than B and less than YB.

[0221] If YN is not N, and either YN is NOGROUP or YB is less than orequal to B,

[0222] Let (ZB, ZN) be the segment following (YB, YN).

[0223] If M+1 is less than ZB, insert a new segment M+1, YN) before (ZB,ZN).

[0224] Replace (YB, YN) by (YB, N).

[0225] Rewriting HTML Resources

[0226] As explained above with reference to FIG. 3 (B5), when areflector or repeater serves a resource which itself includes resourceidentifiers (e.g., a HTML resource), that resource is modified(rewritten) to pre-reflect resource identifiers (URLs) of repeatableresources that appear in the resource. Rewriting ensures that when abrowser requests repeatable resources identified by the requestedresource, it gets them from a repeater without going back to the originserver, but when it requests non-repeatable resources identified by therequested resource, it will go directly to the origin server. Withoutthis optimization, the browser would either make all requests at theorigin server (increasing traffic at the origin server and necessitatingfar more redirections from the origin server), or it would make allrequests at the repeater (causing the repeater to redundantly requestand copy resources which could not be cached, increasing the overhead ofserving such resources).

[0227] Rewriting requires that a repeater has been selected (asdescribed above with reference to the Best Repeater Selector). Rewritinguses a so-called BASE directive. The BASE directive lets the HTMLidentify a different base server. (The base address is normally theaddress of the HTML resource.)

[0228] Rewriting is performed as follows:

[0229] F1. A BASE directive is added at the beginning of the HTMLresource, or modified where necessary. Normally, a browser interpretsrelative URLs as being relative to the default base address, namely, theURL of the HTML resource (page) in which they are encountered. The BASEaddress added specifies the resource at the reflector which originallyserved the resource. This means that unprocessed relative URLs (such asthose generated by Javascrip™ programs) will be interpreted as relativeto the reflector. Without this BASE address, browsers would combinerelative addresses with repeater names to create URLs which were not inthe form required by repeaters (as described above in step D1).

[0230] F2. The rewriter identifies directives, such as embedded imagesand anchors, containing URLs. If the rewriter is running in a reflector,it must parse the HTML file to identify these directives.

[0231] If it is running in a repeater, the rewriter may have access topre-computed information that identifies the location of each URL(placed in the HTML file in step F4).

[0232] F3. For each URL encountered in the resource to be re-written,the rewriter must determine whether the URL is repeatable (as in stepsB1-B2). If the URL is not repeatable, it is not modified. On the otherhand, if the URL is repeatable, it is modified to refer to the selectedrepeater.

[0233] F4. After all URLs have been identified and modified, if theresource is being served to a repeater, a table is appended at thebeginning of the resource that identifies the location and content ofeach URL encountered in the resource. (This step is an optimizationwhich eliminates the need for parsing HTML resources at the repeater.)

[0234] F5. Once all changes have been identified, a new length iscomputed for the resource (page). The length is inserted in the HTTPheader prior to serving the resource.

[0235] An extension of HTML, known as XML, is currently being developed.The process of rewriting URLs will be similar for XML, with somedifferences in the mechanism that parses the resource and identifiesembedded URLs.

[0236] Handling Non-HTTP Protocols

[0237] This invention makes it possible to reflect references toresources that are served by protocols other than HTTP, for instance,the File Transfer Protocol (FTP) and audio/video stream protocols.However, many protocols do not provide the ability to redirect requests.It is, however, possible to redirect references before requests areactually made by rewriting URLs embedded in HTML pages. The followingmodifications to the above algorithms are used to support thiscapability.

[0238] In F4, the rewriter rewrites URLs for servers if those serversappear in a configurable table of cooperating origin server or so-calledco-servers. The reflector operator can define this table to include FTPservers and other servers. A rewritten URL that refers to a non-HTTPresource takes the form:

[0239] http://<repeater>/<originserver>@proxy=<scheme>[:<type>]@/resource

[0240] where <scheme> is a supported protocol name such as “ftp”. ThisURL format is an alternative to the form shown in B3.

[0241] In C3, the repeater looks for a protocol embedded in the arrivingrequest. If a protocol is present and the requested resource is notalready cached, the repeater uses the selected protocol instead of thedefault HTTP protocol to request the resource when serving it andstoring it in the cache.

[0242] System Configuration and Management

[0243] In addition to the processing described above, the repeaternetwork requires various mechanisms for system configuration and networkmanagement. Some of these mechanisms are described here.

[0244] Reflectors allow their operators to synchronize repeater cachesby performing publishing operations. The process of keeping repeatercaches synchronized is described below. Publishing indicates that aresource or collection of resources has changed.

[0245] Repeaters and reflectors participate in various types of logprocessing. The results of logs collected at repeaters are collected andmerged with logs collected at reflectors, as described below.

[0246] Adding Subscribers to the Repeater Network

[0247] When a new subscriber is added to the network, information aboutthe subscriber is entered in a Subscriber Table at the master repeaterand propagated to all repeaters in the network. This informationincludes the Committed Aggregate Information Rate (CAIR) for serversbelonging to the subscriber, and a list of the repeaters that may beused by servers belonging to the subscriber.

[0248] Adding Reflectors to the Repeater Network

[0249] When a new reflector is added to the network, it simply connectsto and announces itself to a contact repeater, preferably using asecurely encrypted certificate including the repeater's subscriberidentifier.

[0250] The contact repeater determines whether the reflector networkaddress is permitted for this subscriber. If it is, the contact repeateraccepts the connection and updates the reflector with all necessarytables (using version numbers to determine which tables are out ofdate).

[0251] The reflector processes requests during this time, but is not“enabled” (allowed to reflect requests) until all of its tables arecurrent.

[0252] Keeping Repeater Caches Synchronized

[0253] Repeater caches are coherent, in the sense that when a change toa resource is identified by a reflector, all repeater caches arenotified, and accept the change in a single transaction.

[0254] Only the identifier of the changed resource (and not the entireresource) is transmitted to the repeaters; the identifier is used toeffectively invalidate the corresponding cached resource at therepeater. This process is far more efficient than broadcasting thecontent of the changed resource to each repeater.

[0255] A repeater will load the newly modified resource the next time itis requested.

[0256] A resource change is identified at the reflector either manuallyby the operator, or through a script when files are installed on theserver, or automatically through a change detection mechanism (e.g., aseparate process that checks regularly for changes).

[0257] A resource change causes the reflector to send an “invalidate”message to its contact repeater, which forwards the message to themaster repeater. The invalidate message contains a list of resourceidentifiers (or regular expressions identifying patterns of resourceidentifiers) that have changed. Regular expressions are used toinvalidate a directory or an entire server.) The repeater network uses atwo-phase commit process to ensure that all repeaters correctlyinvalidate a given resource.

[0258] The invalidation process operates as follows:

[0259] The master broadcasts a “phase 1” invalidation request to allrepeaters indicating the resources and regular expressions describingsets of resources to be invalidated.

[0260] When each repeater receives the phase 1 message, it first placesthe resource identifiers or regular expressions into a list of resourceidentifiers pending invalidation.

[0261] Any resource requested (in C3) that is in the pendinginvalidation list may not be served from the cache. This prevents thecache from requesting the resource from a peer cache which may not havereceived an invalidation notice. Were it to request a resource in thismanner, it might replace the newly invalidated resource by the same, nowstale, data.

[0262] The repeater then compares the resource identifier of eachresource in its cache against the resource identifiers and regularexpressions in the list.

[0263] Each match is invalidated by marking it stale and optionallyremoving it from the cache. This means that a future request for theresource will cause it to retrieve a new copy of the resource from thereflector.

[0264] When the repeater has completed the invalidation, it returns anacknowledgment to the master. The master waits until all repeaters haveacknowledged the invalidation request.

[0265] If a repeater fails to acknowledge within a given period, it isdisconnected from the master repeater. When it reconnects, it will betold to flush its entire cache, which will eliminate any consistencyproblem. (To avoid flushing the entire cache, the master could keep alog of all invalidations performed, sorted by date, and flush only filesinvalidated since the last time the reconnecting repeater successfullycompleted an invalidation. In the presently preferred embodiments thisis not done since it is believed that repeaters will seldom disconnect.)

[0266] When all repeaters have acknowledged invalidation (or timed out)the repeater broadcasts a “phase 2” invalidation request to allrepeaters. This causes the repeaters to remove the correspondingresource identifiers and regular expressions from the list of resourceidentifiers pending invalidation.

[0267] In another embodiment, the invalidation request will be extendedto allow a “server push”. In such requests, after phase 2 of theinvalidation process has completed, the repeater receiving theinvalidation request will immediately request a new copy of theinvalidated resource to place in its cache.

[0268] Logs and Log Processing

[0269] Web server activity logs are fundamental to monitoring theactivity in a Web site. This invention creates “merged logs” thatcombine the activity at reflectors with the activity at repeaters, sothat a single activity log appears at the origin server showing all Webresource requests made on behalf of that site at any repeater.

[0270] This merged log can be processed by standard processing tools, asif it had been generated locally.

[0271] On a periodic basis, the master repeater (or its delegate)collects logs from each repeater. The logs collected are merged, sortedby reflector identifier and timestamp, and stored in a dated file on aper-reflector basis. The merged log for a given reflector represents theactivity of all repeaters on behalf of that reflector. On a periodicbasis, as configured by the reflector operator, a reflector contacts themaster repeater to request its merged logs. It downloads these andmerges them with its locally maintained logs, sorting by timestamp. Theresult is a merged log that represents all activity on behalf ofrepeaters and the given reflector.

[0272] Activity logs are optionally extended with information importantto the repeater network, if the reflector is configured to do so by thereflector operator. In particular, an “extended status code” indicatesinformation about each request, such as:

[0273] 1. request was served by a reflector locally;

[0274] 2. request was reflected to a repeater;*

[0275] 3. request was served by a reflector to a repeater;*

[0276] 4. request for non-repeatable resource was served by repeater;*

[0277] 5. request was served by a repeater from the cache;

[0278] 6. request was served by a repeater after filling cache;

[0279] 7. request pending invalidation was served by a repeater.

[0280] (The activities marked with “*” represent intermediate states ofa request and do not normally appear in a final activity log.)

[0281] In addition, activity logs contain a duration, and extendedprecision timestamps. The duration makes it possible to analyze the timerequired to serve a resource, the bandwidth used, the number of requestshandled in parallel at a given time, and other quite useful information.The extended precision timestamp makes it possible to accurately mergeactivity logs.

[0282] Repeaters use the Network Time Protocol (NTP) to maintainsynchronized clocks. Reflectors may either use NTP or calculate a timebias to provide roughly accurate timestamps relative to their contactrepeater.

[0283] Enforcing Committed Aggregate Information Rate

[0284] The repeater network monitors and limits the aggregate rate atwhich data is served on behalf of a given subscriber by all repeaters.This mechanism provides the following benefits:

[0285] 1. provides a means of pricing repeater service;

[0286] 2. provides a means for estimating and reserving capacity atrepeaters;

[0287] 3. provides a means for preventing clients of a busy site fromlimiting access to other sites.

[0288] For each subscriber, a “threshold aggregate information rate”(TAIR) is configured and maintained at the master repeater. Thisthreshold is not necessarily the committed rate, it may be a multiple ofcommitted rate, based on a pricing policy.

[0289] Each repeater measures the information rate component of eachreflector for which it serves resources, periodically (typically aboutonce a minute), by recording the number of bytes transmitted on behalfof that reflector each time a request is delivered. The table thuscreated is sent to the master repeater once per period. The masterrepeater combines the tables from each repeater, summing the measuredinformation of each reflector over all repeaters that serve resourcesfor that reflector, to determine the “measured aggregate informationrate” (MAIR) for each reflector.

[0290] If the MAIR for a given reflector is greater than the TAIR forthat reflector, the MAIR is transmitted by the master to all repeatersand to the respective reflector.

[0291] When a reflector receives a request, it determines whether itsmost recently calculated MAIR is greater than its TAIR. If this is thecase, the reflector probabilistically decides whether to suppressreflection, by serving the request locally (in B2). The probability ofsuppressing the reflection increases as an exponential function of thedifference between the MAIR and the CAIR.

[0292] Serving a request locally during a peak period may strain thelocal origin server, but it prevents this subscriber from taking morethan allocated bandwidth from the shared repeater network.

[0293] When a repeater receives a request for a given subscriber (inC2), it determines whether the subscriber is running near its thresholdaggregate information rate. If this is the case, it probabilisticallydecides whether to reduce its load by redirecting the request back tothe reflector. The probability increases exponentially as thereflector's aggregate information rate approaches its limit.

[0294] If a request is reflected back to a reflector, a specialcharacter string is attached to the resource identifier so that thereceiving reflector will not attempt to reflect it again. In the currentsystem, this string has the form

[0295] “src=overload”.

[0296] The reflector tests for this string in B2.

[0297] The mechanism for limiting Aggregate Information Rate describedabove is fairly coarse. It limits at the level of sessions with clients(since once a client has been reflected to a given repeater, therewriting process tends to keep the client coming back to that repeater)and, at best, individual requests for resources. A more fine-grainedmechanism for enforcing TAIR limits within repeaters operates byreducing the bandwidth consumption of a busy subscriber when othersubscribers are competing for bandwidth.

[0298] The fine-grained mechanism is a form of data “rate shaping”. Itextends the mechanism that copies resource data to a connection when areply is being sent to a client. When an output channel is establishedat the time a request is received, the repeater identifies whichsubscriber the channel is operating for, in C2, and records thesubscriber in a data field associated with the channel. Each time a“write” operation is about to be made to the channel, the Metered OutputStream first inspects the current values of the MAIR and TAIR,calculated above, for the given subscriber. If the MAIR is larger thanthe TAIR, then the mechanism pauses briefly before performing the writeoperation. The length of the pause is proportional to the amount theMAIR exceeds the TAIR. The pause ensures that tasks sending otherresources to other clients, perhaps on behalf of other subscribers, willhave an opportunity to send their data.

[0299] Repeater Network Resilience

[0300] The repeater network is capable of recovering when a repeater ornetwork connection fails.

[0301] A repeater cannot operate unless it is connected to the masterrepeater. The master repeater exchanges critical information with otherrepeaters, including information about repeater load, aggregateinformation rate, subscribers, and link cost.

[0302] If a master fails, a “succession” process ensures that anotherrepeater will take over the role of master, and the network as a wholewill remain operational. If a master fails, or a connection to a masterfails through a network problem, any repeater attempting to communicatewith the master will detect the failure, either through an indicationfrom TCP/IP, or by timing out from a regular “heartbeat” message itsends to the master.

[0303] When any repeater is disconnected from its master, it immediatelytries to reconnect to a series of potential masters based on aconfigurable file called its “succession list”.

[0304] The repeater tries each system on the list in succession until itsuccessfully connects to a master. If in this process, it comes to itsown name, it takes on the role of master, and accepts connections fromother repeaters. If a repeater which is not at the top of the listbecomes the master, it is called the “temporary master”.

[0305] A network partition may cause two groups of repeaters each toelect a master. When the partition is corrected, it is necessary thatthe more senior master take over the network. Therefore, when a repeateris temporary master, it regularly tries to reconnect to any master aboveit in the succession list. If it succeeds, it immediately disconnectsfrom all of the repeaters connected to it. When they retry theirsuccession lists, they will connect to the more senior master repeater.

[0306] To prevent losses of data, a temporary master does not acceptconfiguration changes and does not process log files. In order to takeon these tasks, it must be informed that it is primary master by manualmodification of its successor list. Each repeater regularly reloads itssuccessor list to determine whether it should change its idea of who themaster is.

[0307] If a repeater is disconnected from the master, it mustresynchronize its cache when it reconnects to the master. The master canmaintain a list of recent cache invalidations and send to the repeaterany invalidations it was not able to process while disconnected. If thislist is not available for some reason (for instance, because thereflector has been disconnected too long), the reflector must invalidateits entire cache.

[0308] A reflector is not permitted to reflect requests unless it isconnected to a repeater. The reflector relies on its contact repeaterfor critical information, such as load and Link Cost Tables, and currentaggregate information rate. A reflector that is not connected to arepeater can continue to receive requests and handle them locally.

[0309] If a reflector loses its connection with a repeater, due to arepeater failure or network outage, it continues to operate while ittries to connect to a repeater.

[0310] Each time a reflector attempts to connect to a repeater, it usesDNS to identify a set of candidate repeaters given a domain name thatrepresents the repeater network. The reflector tries each repeater inthis set until it makes a successful contact. Until a successful contactis made, the reflector serves all requests locally. When a reflectorconnects to a repeater, the repeater can tell it to attempt to contact adifferent repeater; this allows the repeater network to ensure that noindividual repeater has too many contacts.

[0311] When contact is made, the reflector provides the version numberof each of its tables to its contact repeater. The repeater then decideswhich tables should be updated and sends appropriate updates to thereflector. Once all tables have been updated, the repeater notifies thereflector that it may now start reflecting requests.

[0312] Using a Proxy Cache within a Repeater

[0313] Repeaters are intentionally designed so that any proxy cache canbe used as a component within them. This is possible because therepeater receives HTTP requests and converts them to a form recognizedby the proxy cache.

[0314] On the other hand, several modifications to a standard proxycache have been or may be made as optimizations. This includes, inparticular, the ability to conveniently invalidate a resource, theability to support cache quotas, and the ability to avoid making anextra copy of each resource as it passes from the proxy cache throughthe repeater to the requester.

[0315] In a preferred embodiment, a proxy cache is used to implement C3.The proxy cache is dedicated for use only by one or more repeaters. Eachrepeater requiring a resource from the proxy cache constructs a proxyrequest from the inbound resource request. A normal HTTP GET request toa server contains only the pathname part of the URL—the scheme andserver name are implicit. (In an HTTP GET request to a repeater, thepathname part of the URL includes the name of the origin server onbehalf of which the request is being made, as described above.) However,a proxy agent GET request takes an entire URL. Therefore, the repeatermust construct a proxy request containing the entire URL from the pathportion of the URL it receives. Specifically, if the incoming requesttakes the form:

[0316] GET/<origin server>/<path>

[0317] then the repeater constructs a proxy request of the form:

[0318] GET http://<origin server>/<path>

[0319] and if the incoming request takes the form:

[0320] GET <origin server>@proy=<scheme>:<type>@/<path>

[0321] then the repeater constructs a proxy request of the form:

[0322] GET <scheme>://<origin server>/<path>

[0323] Cache Control

[0324] HTTP replies contain directives called cache control directives,which are used to indicate to a cache whether the attached resource maybe cached and if so, when it should expire. A Web site administratorconfigures the Web site to attach appropriate directives. Often, theadministrator will not know how long a page will be fresh, and mustdefine a short expiration time to try to prevent caches from servingstale data. In many cases, a Web site operator will indicate a shortexpiration time only in order to receive the requests (or hits) thatwould otherwise be masked by the presence of a cache. This is known inthe industry as “cache-busting”. Although some cache operators mayconsider cache-busting to be impolite, advertisers who rely on thisinformation may consider it imperative.

[0325] When a resource is stored in a repeater, its cache directives canbe ignored by the repeater, because the repeater receives explicitinvalidation events to determine when a resource is stale. When a proxycache is used as the cache at the repeater, the associated cachedirectives may be temporarily disabled. However, they must be re-enabledwhen the resource is served from the cache to a client, in order topermit the cache-control policy (including any cache-busting) to takeeffect.

[0326] The present invention contains mechanisms to prevent the proxycache within a repeater from honoring cache control directives, whilepermitting the directives to be served from the repeater.

[0327] When a reflector serves a resource to a repeater in B4, itreplaces all cache directives by modified directives that are ignored bythe repeater proxy cache. It does this by prefixing a distinctive stringsuch as “wr-” to the beginning of the HTTP tag. Thus, “expires” becomes“wr-expires”, and “cache-control” becomes “wr-cache-control”. Thisprevents the proxy cache itself from honoring the directives. When arepeater serves a resource in C4, and the requesting client is notanother repeater, it searches for HTTP tags beginning with “wr-” andremoves the “wr-”. This allows the clients retrieving the resource tohonor the directives.

[0328] Resource Revalidation

[0329] There are several cases where a resource may be cached so long asthe origin server is consulted each time it is served. In one case, therequest for the resource is attached to a so-called “cookie”. The originserver must be presented with the cookie to record the request anddetermine whether the cached resource may be served or not. In anothercase, the request for the resource is attached to an authenticationheader (which identifies the requester with a user id and password).Each new request for the resource must be tested at the origin server toassure that the requester is authorized to access the resource.

[0330] The HTTP 1.1 specification defines a reply header titled“Must-Revalidate” which allows an origin server to instruct a proxycache to “revalidate” a resource each time a request is received.Normally, this mechanism is used to determine whether a resource isstill fresh. In the present invention, Must-Revalidate makes it possibleto ask an origin server to validate a request that is otherwise servedfrom a repeater.

[0331] The reflector rule base contains information that determineswhich resources may be repeated but must be revalidated each time theyare served. For each such resource, in B4, the reflector attaches aMust-Revalidate header. Each time a request comes to a repeater for acached resource marked with a Must-Revalidate header, the request isforwarded to the reflector for validation prior to serving the requestedresource.

[0332] Cache Quotas

[0333] The cache component of a repeater is shared among thosesubscribers that reflect clients to that repeater. In order to allowsubscribers fair access to storage facilities, the cache may be extendedto support quotas.

[0334] Normally, a proxy cache may be configured with a disk spacethreshold T. Whenever more than T bytes are stored in the cache, thecache attempts to find resources to eliminate.

[0335] Typically a cache uses the least-recently-used (LRU) algorithm todetermine which resources to eliminate; more sophisticated caches useother algorithms. A cache may also support several threshold values—forinstance, a lower threshold which, when reached, causes a low prioritybackground process to remove items from the cache, and a higherthreshold which, when reached, prevents resources from being cacheduntil sufficient free disk space has been reclaimed.

[0336] If two subscribers A and B share a cache, and more resources ofsubscriber A are accessed during a period of time than resources ofsubscriber B, then fewer of B's resources will be in the cache when newrequests arrive. It is possible that, due to the behavior of A, B'sresources will never be cached when they are requested. In the presentinvention, this behavior is undesirable. To address this issue, theinvention extends the cache at a repeater to support cache quotas.

[0337] The cache records the amount of space used by each subscriber inD_(S), and supports a configurable threshold T_(S) for each subscriber.

[0338] Whenever a resource is added to the cache (at C3), the valueD_(S) is updated for the subscriber providing the resource. If D_(S) islarger than T_(S), the cache attempts to find resources to eliminate,from among those resources associated with subscriber S. The cache iseffectively partitioned into separate areas for each subscriber.

[0339] The original threshold T is still supported. If the sum ofreserved segments for each subscriber is smaller than the total spacereserved in the cache, the remaining area is “common” and subject tocompetition among subscribers.

[0340] Note, this mechanism might be implemented by modifying theexisting proxy cache discussed above, or it might also be implementedwithout modifying the proxy cache—if the proxy cache at least makes itpossible for an external program to obtain a list of resources in thecache, and to remove a given resource from the cache.

[0341] Rewriting from Repeaters

[0342] When a repeater receives a request for a resource, its proxycache may be configured to determine whether a peer cache contains therequested resource. If so, the proxy cache obtains the resource from thepeer cache, which can be faster than obtaining it from the origin server(the reflector). However, a consequence of this is that rewritten HTMLresources retrieved from the peer cache would identify the wrongrepeater. Thus, to allow for cooperating proxy caches, resources arepreferably rewritten at the repeater.

[0343] When a resource is rewritten for a repeater, a special tag isplaced at the beginning of the resource. When constructing a reply, therepeater inspects the tag to determine whether the resource indicatesthat additional rewriting is necessary. If so, the repeater modifies theresource by replacing references to the old repeater with references tothe new repeater.

[0344] It is only necessary to perform this rewriting when a resource isserved to the proxy cache at another repeater.

[0345] Repeater-Side Include

[0346] Sometimes, an origin server constructs a custom resource for eachrequest (for instance, when inserting an advertisement based on thehistory of the requesting client). In such a case, that resource must beserved locally rather than repeated. Generally, a custom resourcecontains, along with the custom information, text and references toother, repeatable, resources.

[0347] The process that assembles a “page” from a text resource andpossibly one or more image resources is performed by the Web browser,directed by HTML. However, it is not possible using HTML to cause abrowser to assemble a page using text or directives from a separateresource. Therefore, custom resources often necessarily contain largeamounts of static text that would otherwise be repeatable.

[0348] To resolve this potential inefficiency, repeaters recognize aspecial directive called a “repeater side include”. This directive makesit possible for the repeater to assemble a custom resource, using acombination of repeatable and local resources. In this way, the statictext can be made repeatable, and only the special directive need beserved locally by the reflector.

[0349] For example, a resource X might consist of custom directivesselecting an advertising banner, followed by a large text article. Tomake this resource repeatable, the Web site administrator must break outa second resource, Y, to select the banner. Resource X is modified tocontain a repeater-side include directive identifying resource Y, alongwith the article. Resource Y is created and contains only the customdirectives selecting an ad banner. Now resource X is repeatable, andonly resource Y, which is relatively small, is not repeatable.

[0350] When a repeater constructs a reply, it determines whether theresource being served is an HTML resource, and if so, scans it forrepeater-side include directives. Each such directive includes a URL,which the repeater resolves and substitutes in place of the directive.The entire resource must be assembled before it is served, in order todetermine its final size, as the size is included in a reply headerahead of the resource.

[0351] Thus, a method and apparatus for dynamically replicating selectedresources in computer networks is provided. One skilled in the art willappreciate that the present invention can be practiced by other than thedescribed embodiments, which are presented for purposes of illustrationand not limitation, and the present invention is limited only by theclaims that follow.

What is claimed:
 1. A method of processing resource requests in acomputer network, the method comprising, (i) by a client: (A) making arequest for a particular resource from an origin server, the requestincluding a resource identifier for the particular resource; (ii) by areflector: (B) intercepting the request from the client to the originserver; (C) selecting a repeater to process the request; (D) providingto the client a modified resource identifier designating the repeater;*iii) by the client: (E) receiving the modified resource identifier fromthe reflector; and (F) making a request for the particular resource fromthe repeater designated in the modified resource identifier; (iv) by therepeater: (G) receiving the request from the client; and (H) returningthe requested resource to the client.
 2. A method as in claim 1 furthercomprising, by the repeater: (I) making a request for the resource fromthe origin server; and (J) receiving the resource from the originserver.
 3. A method as in claim 1 wherein the selecting of a repeater bythe reflector comprises: (C1) partitioning the network into groups; (C2)determining which group the client is in; (C3) selecting, from aplurality of repeaters in the network, a set of repeaters having alowest cost relative to the group which the client is in; and (C4)selecting as the repeater a member of the selected set of repeaters. 4.A method as in claim 3 , wherein the cost of a repeater is a value basedon that repeater's current load and a maximum load for that repeater. 5.A method as in claim 3 , wherein the cost of a repeater is a value basedon a predicted cost or speed of transmission between the repeater and aclient in the group.
 6. A method as in claim 1 wherein the particularresource itself contains at least one other resource identifier of atleast one other resource, the method further comprising: rewriting theparticular resource to replace at least some of the resource identifierscontained therein with modified resource identifiers designating arepeater instead of the origin server.
 7. A method as in claim 6 whereinthe rewriting is performed by one of the repeater, the reflector oranother repeater.
 8. A method of processing resource requests in acomputer network, the method comprising, (i) by a client (A) making arequest for a particular resource from an origin server, the requestincluding a resource identifier for the particular resource; (ii) by areflector: (B) intercepting the request from the client to the originserver; (C) determining whether to reflect the request to a repeater;(D) when the reflector determines not to reflect the request, forwardingthe request to the origin server, otherwise (D1) selecting a repeater toprocess the request; (D2) providing to the client a modified resourceidentifier designating the repeater.
 9. A method as in claim 8 , furthercomprising, when the reflector determines to reflect the request, (iii)by the client: (E) receiving the modified resource identifier from thereflector; and (F) making a request for the particular resource from therepeater designated in the modified resource identifier; (iv) by therepeater: (G) receiving the request from the client; and (H) returningthe requested resource to the client.
 10. A method as in claim 8 whereinthe reflector determines whether to reflect a request by comparing theresource identifier with regular expression patterns of repeatableresources.
 11. A method as in claim 8 , wherein the reflector has athreshold aggregate information rate (TAIR) associated therewith, andwherein the determining of whether to reflect the request to a repeatercomprises: determining whether the TAIR of the reflector is exceeded bya measured aggregate information rate (MAIR) for the reflector, whereinthe reflector determines not to reflect the request when the MAIRexceeds the TAIR for the reflector.
 12. A method as in claim 8 , whereinthe reflector has a threshold aggregate information rate (TAIR)associated therewith, and wherein the determining of whether to reflectthe request to a repeater comprises: probabilistically determiningwhether the TAIR of the reflector is exceeded by a measured aggregateinformation rate (MAIR) for the reflector, wherein the reflectordetermines not to reflect the request as an exponential function of thedifference between the MAIR and the TAIR.
 13. A method as in any ofclaims 11-12, wherein the MAIR is obtained from repeaters according tothe rate at which they have transmitted data on behalf of the reflectorduring a given time interval.
 14. A method as in any one of claims 1-12wherein the network is the Internet and wherein the resource identifieris a uniform resource locator (URL) for designating resources on theInternet, and wherein the modified resource identifier is a URLdesignating the repeater and indicating the reflector or origin server,and wherein the modified resource identifier is provided to the clientusing a REDIRECT message.
 15. In a computer network wherein clientsrequest resources from origin servers, a method comprising: providing atleast one repeater; providing reflectors at some of the origin servers,each reflector intercepting client resource requests made to itsrespective origin server; and each reflector selectively redirectingclient resource requests for certain resources to one of the repeaters.16. A method as in claim 15 further comprising, by repeaters in thenetwork: servicing redirected client resource requests; and selectivelymaintaining copies of requested resources, whereby resourcescorresponding to redirected resource requests are selectively migratedfrom their origin servers to one or more repeaters.
 17. A computernetwork comprising: a plurality of origin servers, at least some of theorigin servers having reflectors associated therewith; a plurality ofrepeaters; and a plurality of clients, wherein each reflector is adaptedto intercept resource requests made to its respective origin server andto selectively redirect the resource requests to a dynamically selectedrepeater.
 18. In a computer network wherein clients request resourcesfrom origin servers, a reflector mechanism associated with an originserver, the reflector mechanism comprising: means for intercepting aresource request made by client of an origin server; means for analyzingthe resource request to determine whether to service the request locallyat the origin server; means for determining a best repeater in thenetwork to service the request when the analyzing means determines thatthe request should not be serviced locally; and means for redirectingthe client to the best repeater.
 19. A reflector mechanism as in claim18 wherein the network is partitioned into groups and the means fordetermining the best repeater comprises: means for determining whichgroup the client is in; means for selecting, from a plurality ofrepeaters in the network, a set of repeaters having a lowest costrelative to the group the client is in; and means for selecting as thebest repeater a member of the set of repeaters.
 20. A reflectormechanism as in claim 19 , wherein the cost of a repeater is a valuebased on a predicted cost or speed of transmission between the repeaterand a client in the group.
 21. A mechanism as in claim 19 , wherein thecost of a repeater is a value based on that repeaters current load and amaximum load for that repeater.
 22. A reflector as in claim 16 whereinthe resource itself contains resource identifiers, the reflector furthercomprising: means for rewriting the resource to replace at least some ofthe resource identifiers contained therein with modified resourceidentifiers designating the repeater instead of the origin server. 23.In a computer network wherein clients request resources from originservers, a repeater mechanism comprising: means for receiving a resourcerequest from a client; means for determining whether the resource isavailable locally; means for, when it is determined that the resource isnot available locally, obtaining the resource from an origin server; andmeans for providing the resource to the client.
 24. A reflector as inclaim 18 wherein the resource itself contains resource identifiers, therepeater further comprising: means for rewriting the resource to replaceat least some of the resource identifiers contained therein withmodified resource identifiers designating the repeater instead of theorigin server.